The dominant general purpose computing architectures and system implementations for business, science, engineering and other applications rarely offer long-term, complete concepts and solutions for organizing, storing, and manipulating data on external storage devices with a view on an application's entire life-cycle. In general, support for active use of data is rather limited to a historically relatively short time period, rarely ever beyond the scope of one to two years. Control and manipulation of the storage hardware components have become, nearly exclusively, a matter of the particular operating system and its device handling services in control of any specific computing platform. Based on these fundamental device services, higher software layers most often provide concepts and functions for applications to straightforwardly handle externally stored data. A few examples of such data from a very large field of applications are: payment orders, airline passenger reservations, seismic signals, engineering and construction data, business reports, and weather forecast data.
Storage hardware considerations are not part of this description for the following reasons: The application relevant storage software functions are shielded from hardware characteristics either through complete hardware function transparency, or the hardware characteristics are treated on a very low software level and are, therefore, not visible for higher software layers near the application, or the application itself. This applies, for example, to modern RAID (Redundant Array of Inexpensive Disks) implementations providing a high protection level against loss of data when one disk drive fails. It also applies to data striping techniques where data strings to be written to or read from a disk are cut into many small pieces which are, in turn, written to and read from separate devices in parallel.
The vast majority of the mentioned higher software layers for the handling of externally stored data are either commercially available as products, or are part of an operating system and provided together with it, or are available from the public software domain. In principle, they apply to nearly all of the presently installed and known computing systems, covering likewise, to give some examples, high-end parallel supercomputers, large business computing systems, commercial and engineering midrange systems, and workstations down to simple personal computers. On all these platforms, storage devices with direct access capability form the backbone of most modern applications for the processing of externally stored data. Consequently, data handling software with the capability to exploit such direct access storage devices is a key component in the overall design and implementation of modern applications. Data are stored and accessed by application programs through the use of the following two main categories of data handling software:
1. Data Base Management Systems (DBMS), especially those which implement the concept of entity-relationships (usually named relational) by offering tabular data structures, are prevalent. The data manipulation language of Relational Data Base Management Systems (RDBMS) is based on the mathematical foundations of the set theory and enables both set processing and data contents (values) based operations. Recent enhancements were provided in the areas of abstract data typing, built-in functions with trigger levels, and multimedia support. These enhancements partially implement concepts from the object-oriented paradigm. A widely used RDBMS product set is IBM's Database2 (DB2) family offered on all of its major computing platforms. PA1 2. File Systems are historically older but still in widespread use. They offer less functions to their using applications than DBMS or RDBMS, and are more demanding to deal with from a designer's or programmer's perspectives in order to achieve the same objectives. Therefore, they are no longer the primary choice in the component selection process of application development. However, the fact that File Systems or core functions of them are quite commonly used for building the higher level DBMS and RDBMS, has two important consequences: File Systems will further be used in spite of their diminishing role, and their architectural and implementation limitations manifest themselves indirectly in limitations of their using DBMS/RDBMS. An example of a well known File System is IBM's Virtual Storage Access Method (VSAM) on its /360, /1370, and /390 types of computers.
All known implementations limit the size of individual data clusters grouped together and commonly referred to as files, or access paths as they are called in IBM's midrange computer family AS/400. The maximum file size is given by the size of a File System's addressing scheme: 32 bits span a range of up to 4 Giga bytes. The size of 32 bits is a meaningful trade-off between space occupied by the addresses in main (internal) and external memories and the typical size of data clusters to be stored. In addition, 32 bits well fit most computer systems' hardware structures, and enable an efficient runtime behavior. Some Database Systems accommodate larger, but nevertheless limited, sizes of data to be stored as a single entity. For example, DB2's maximum table size on an MVS/ESA computing platform is currently 64 Giga bytes. The step beyond an individual file limitation of 4 GB is done by mapping the data to a fixed number of partitions. As individual partitions can be recognized and recovered independently, this concept overcomes the problem of excessive run-times for data recovery and reorganization. However, the base for any partition is still a single file, limited by the maximum size imposed by the underlying File System. Data are mapped to these partitions based on the value of a key, either via a pre-established and fixed assignment of key ranges (as is done by DB2 on IBM's large business computers with MVS/ESA as an operating system), or through a hashing algorithm based on key values (as implemented by DB2 Parallel Edition for the Scalable POWERparallel computers based on IBM's Unix-derivative, AIX/6000). Changing the organization of partitions implies an unload and reload of the entire database.
The fact that individual database components (e.g., tables) are limited in their maximum size fits especially ill with very large collections of data accumulated with time. Either new data are added according to keyrange definitions to the appropriate partition; then the mapping to the pre-established partitions has to be adjusted before the most heavily loaded partition fills up. Or, new data are spread via a hashing algorithm more or less equally over all existing partitions. Then the critical point of completely filled up partitions will occur in all partitions at about the same time.
Both approaches, i.e. partitions with keyrange affinity and hashing algorithm driven solutions require a complete database unload/reload before increasing the number of partitions and to redefine the assigned keyranges or hashing parameters. But both approaches suffer from maximum upper limitations of size and number of partitions. The frequency of such data reorganizations (unload/reload) affects the overall performance and availability. Such reorganizations are characterized by (1) the need for an exclusive access to the data involved during the entire reorganization process, excluding, hence, applications from accessing data during this time, and (2) by considerable time periods required for such reorganizations, often in the range of many hours or beyond. These shortcomings of present File and Database Systems for very large data collections have to be overcome by people-intensive administrative space management procedures and, on the level of the individual application, by considerable skilled human resources and calendar time spent on these issues during the initial development and later maintenance processes.
Another disadvantage of current Database Systems is that collections of data to be stored over very long periods of time have to remain stored, with all of their partitions, under the control of one and the same instance of a DBMS/RDBMS, or File System. This prevents the fast and easy exploitation of new software technologies for new data with long-living data clusters.